Handling Disagreement in Hate Speech Modelling

Abstract Hate speech annotation for training machine learning models is an inherently ambiguous and subjective task. In this paper, we adopt a perspectivist approach to data annotation, model evaluation hate classification. We first focus on the process argue that it drastically influences final quality. then present three large datasets incorporate annotator disagreement use them train evaluate models. As main point, propose through lens of by applying proper performance measures both annotators’ agreement models’ further poses intrinsic limits achievable When comparing annotators, observed they achieve consistent levels across datasets. reflect upon our results some methodological ethical considerations can stimulate ongoing discussion modelling classification with disagreement.